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DOCUMENT- IDENTIFIER: US 6669482 Bl 

TITLE: Method for teaching interpretative skills in radiology with standardized 
terminology 

Abstract Text (1) : 

A training method for improving diagnostic accuracy and reducing variability in the 
interpretation of radiologic exams. Exam images, whether presented on radiographic 
film, on photographic paper, or on a computer monitor, are interpreted using 
standardized feature descriptors, such as in BI-RADS. The exam interpreter uses the 
feature descriptors to characterize findings in exam images and provides an 
assessment and conclusion concerning the presence of biologic processes that would 
have caused the image finding. Evidence for the presence or absence of these 
biologic processes is obtained. Measures of diagnostic accuracy are calculated from 
these data and all this information is tracked and associated with the exams. 
Subsequent to image interpretation, exam findings and tracked data are reviewed in 
a case-based manner to teach the image interpreter to better understand the 
relationship between features and the biologic processes that they are associated 
with, thereby improving the interpreter's diagnostic accuracy and reducing 
variability in their characterization of image findings. Repetitive use of this 
method further improves diagnostic accuracy and reduces variability in the 
characterization of image findings. 

Brief Summary Text (3) : 

The invention relates to a method for improving a radiologists or other 
interpreters skills in interpreting radiologic images whether film images viewed on 
a multiviewer or the like, or digital images viewed on a computer screen. The 
method includes use of standardized terminology during interpretation of radiologic 
studies to characterize image findings. The method tracks an interpreter's use of 
this terminology in addition to their assessments and conclusions concerning the 
presence or absence of biological processes that cause the image finds and tracks 
evidence for the presence or absence of these biological processes. The method 
further employs repetitive feedback to interpreters concerning their diagnostic 
accuracy and use of standardized feature descriptors to refine their understanding 
of the appropriate use of the terminology. 

Brief Summary Text (5) : 

Variability in the interpretation of radiological exams is a known phenomenon, and 
has recently received considerable attention in the area of breast imaging . For 
example, recent research has demonstrated marked variability in radiologists 1 
interpretation of mammograms (Elmore, 1995 & 1998; Beam, 1996) . The authors of 
these studies have noted the need for efforts to improve accuracy and reduce 
variability in the interpretation of mammograms. Unlike analytic tests (e.g. serum 
electrolytes), the interpretation of radiologic tests involves a significant amount 
of subjective judgment. This phenomenon has been discussed in numerous publications 
(Elmore, 1995; Beam, 1996), and includes both (a) failures in detection (i.e., 
failure to identify an abnormality), and (b) failures of characterization (e.g., 
failure to properly classify an abnormality as benign or malignant once its 
features have been assessed) . The method of the present invention addresses this 
second source of variability, focusing on a method for improving diagnostic 
accuracy and reducing the variability of an interpreters 1 characterization of 
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abnormalities seen on radiologic examinations. 
Brief Summary Text (7) : 

Breast imaging is one of the first subspecialties of radiology to embrace 
standardized reporting. Standardized reporting uses defined terminology and report 
formats to improve consistency and reduce confusion in the reporting of image 
findings and abnormalities. Mammography is the first area of breast imaging in 
which widespread use of standardized reporting is becoming common practice. This 
results, in part, from Federal regulations which went into effect Apr. 28, 1999, 
requiring all mammographic facilities in the United States to use standardized 
assessment and recommendation terminology at the end of mammographic reports. The 
assessment and recommendation language is nearly identical to that used in the 
American College of Radiology 1 s (ACR ! s) Breast Imaging and Reporting Data System 
(BI-RADS) . BI-RADS was developed for standardized mammography reporting and was 
first released in 1993 (Kopans, 1993). The ACR's promotion of BI-RADS helped 
influence the Food and Drug Administration's ( FDA 1 s ) requirement that standardized 
assessment and recommendation terminology appear at the end of mammographic 
reports. The promotion of BI-RADS has also prompted development of standardized 
terminology for other imaging modalities. For example, standardized reporting 
terminology for breast ultrasound is being pursued by several groups (ACR Bulletin 
1999; Hawkins, 1998) . 

Brief Summary Text (9) : 

Standardized reporting formats include lexicons of feature descriptors used to 
categorize image findings and abnormalities (Kopans, 1993; D'Orsi, 1995 & 1998; 
Hawkins, 1998). In the case of BI-RADS, D'Orsi, et al (1993) have attempted to 
group BI-RADS lexicon features according to the probability of their association 
with malignancy. However, it is only recently that the association of BI-RADS 
features with benign and malignant breast disease has been empirically evaluated 
(Lieberman, 1998) . New and/or altered descriptors that better discriminate between 
benign and malignant breast disease will be incorporated into BI-RADS as they are 
discovered. As these type of improvements are made in BI-RADS, proper use of the 
feature descriptors will help guide radiologists to more accurate characterization 
of mammographic findings. The same is anticipated for feature descriptors of other 
standardized reporting systems. 

Brief Summary Text (13) : 

Practice audits in breast imaging have been used for a number of years to improve 
the skills of interpreters. Hence the Agency for Health Care Policy and Research 
(AHCPR) has strongly encouraged them (Basset, 1994), and the AHCPR audit 
recommendations became Federal Law in 1999 (Federal Register, 1998) . Breast imaging 
facilities are now required to track mammography assessments and recommendations 
according to structured assessment and recommendation categories. This aides 
practice in calculating profiles such as true positive, true negative, false 
positive and false negative rates, as well as sensitivity, specificity and positive 
predictive values. Audits containing this information have been shown to be a 
powerful educational tool for refining radiologist 1 s interpretive skills (Bird, 
1992; Sickles, 1990; Spring, 1991; Linver 1992) . However, this type of audit 
information only provides radiologists with a general overview of the strengths and 
weaknesses of their interpretive skills. For example, these audits enable 
radiologists to identify poor specificity in mammographic interpretations. They do 
not provide radiologists with mechanisms to examine the relationship between 
features of image findings and diagnostic decision making. The method of the 
current invention provides this type of mechanism, and like a practice audit, is a 
powerful educational tool. 

Brief Summary Text (15) : 

The primary object of the present invention is a training method to improve the 
accuracy and reduce the variability of anyone who reads and interprets radiologic 
examinations. The method tracks the reader's ( image interpreter's) diagnostic 
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accuracy and use of standardized feature descriptors during interpretation of 
radiologic examinations. The method is not only useful for training readers how to 
appropriately use descriptors of a standardized reporting system, but it also leads 
to a detailed understanding of the association of findings with specific types of 
pathology (e.g., benign and malignant disease). As described herein, the method 
utilizes repetitive feedback concerning an interpreter's diagnostic accuracy and 
use of standardized terminology during exam interpretation. It improves accuracy of 
interpretations and reduces variability in the use of standardized terminology. 

Brief Summary Text (25) : 

In accordance with the invention, generally stated, a training method is described 
by which a radiologist 1 s or other exam interpreter's ability to interpret 
radiologic studies of a patient, whether presented on film, or in a digital format 
is measured. Initially, the radiologist or image interpreter views and interprets a 
set of radiologic exams. For each viewed image in an examination, a finding is made 
and the radiologist or image interpreter describes the features of the finding 
using BI-RADS descriptors. An assessment of the presence of a malignancy is also 
provided. The results are then reviewed to assess both the accuracy of the 
diagnosis and the use of the descriptors. After this initial image interpretation, 
the radiologist or image interpreter reviews their diagnostic accuracy and use of 
feature descriptors, in addition to patient outcomes. Subsequent image 
interpretation and case review aids the radiologist or image interpreter in 
improving his or her proficiency in diagnosis and use of feature descriptors. Other 
objects and features will be in part apparent and in part pointed out hereinafter. 

Detailed Description Text (2) : 

While the method of the invention is for training in the interpretation of 
mammograms using the standardized feature descriptors contained in BI-RADS, the 
method is applicable to any radiologic imaging modality which is interpreted using 
a standardized reporting system. The fundamental objective of the method is both to 
reduce variability in the use of descriptors and to improve diagnostic accuracy in 
mammographic interpretation. 

Detailed Description Text (22) : 

Image interpretation is performed in a mammography reading room, or an environment 
that closely simulates one. This can be a dark, quiet room with a mammography 
viewbox or mammography film multiviewer, or a computer monitor or screen on which 
the images are displayed. Interpreters can use instruments normally available to 
them during interpretation of mammograms. Where film is used, these include; for 
example, a magnifying lens, a bright light for viewing overexposed areas of film, 
and a ruler. For digital images, a graphic user interface, allows the test subject 
to navigate through the images . Where the images are available in various degrees 
of resolution, the subject can move from level of resolution to another as 
appropriate . 

Detailed Description Text (39) : 

References ACR Bulletin 1999, 55(6): 14-19. Working Group on Novel Breast 
Ultrasound Technology. Baker, J A, Kornguth, P., Floyd, C E. Breast imaging 
reporting and data system standardized mammography lexicon: Observer variability in 
lesion description. AJR 1996; 166:773-778. Bassett L W, Hendrick R E, Bassford T L, 
et. al. Quality Determinants of Mammography. AHCPR publication No. 95-0632, 
Rockville Md., 1994. Beam C A, Layde P M, Sullivan D C. Variability in the 
interpretation of screening mammograms by U.S. radiologists . Archives of Internal 
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Bassett L W, Feig S A, Jackson, V P, Kopans, D B, Linver, M N, Sickles E A, 
Stelling, C B. The American College of Radiology Breast Imaging Reporting and Data 
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interpretations of mammograms. NEJM 1995; 331:1493-1499. Elmore J G, Barton M B, 
Moceri V M, Polk S, Arena P J, Fletcher S W. Ten-year risk of false positive 
screening mammograms and clinical breast examinations. NEJM 1998, 338 ( 16) : 1089-96 . 
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R D, Key C R, Stidley C A, King W V. Improvement in mammography interpretation 
skills in a community radiology practice after dedicated teaching courses: 2-year 
medical audit of 38,633 cases. Radiology 1992;184:39-43. Metz C E. ROC methodology 
in radiologic imaging . Investigative Radiology 1986; 21: 720-733. Orel S G, 
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Radiology 1997, 205 (P) : 447 Sickles E A, Ominsky S H, Sollitto R A, Galvin H B, 
Monticciolo D L. Medical audit of a rapid-throughput mammography screening 
practice: methodology and results of 27,114 examinations. Radiology 1990; 175:323- 
327. Shile, P E, Hawkins, H H, O'Neill, M A, Pilgram, T K. Observer Variability in 
use of Terminology of the American College of Radiology (ACR) Breast Imaging and 
Reporting Data System (BI-RADS). Academic Radiology 1997; 4(12): 850. Spring, D B, 
Kimbrell-Wilmot K. Evaluating the success of mammography at the local level: how to 
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Other Reference Publication (4) : 

Breast Cancer Screening/Prevention. Imaginis. Retrieved from the Internet [Mar. 20, 
2 002] URL : < http : //www . pinnacleimaging . com . bresthealth/mqsa . asp?mode=l>. * 
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CLAIMS : 

1. A method for training radiologists and other interpreters of radiologic studies 
by tracking their use of feature descriptors and their diagnostic accuracy in the 
interpretation of radiologic examinations, as well as the health outcome of 
patients on whom the interpreted examinations are obtained, the method comprising: 
tracking each radiologist 1 s characterization of findings in radiologic examinations 
they interpret using accepted feature descriptors; tracking each radiologist 1 s 
assessments and conclusions concerning the presence or absence of biological 
processes that cause the findings they characterize in radiologic examinations; 
tracking evidence for the presence or absence of biologic processes in patients on 
whom the interpreted examinations have been obtained; and, having each radiologist 
review tracked data and associated radiologic exams for the purpose of teaching the 
radiologist to better understand the relationship between feature descriptors and 
the biologic processes imaged in radiologic examinations, thereby improving the 
radiologist 1 s diagnostic accuracy and reducing the variability in their use of 
standardized terminology. 

2. The method of claim 1 further including having the radiologist periodically 
review tracked data and images to determine whether the radiologist's diagnostic 
accuracy improves and the radiologst's variability in the use of standardized 
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terminology decreases. 

26. A method for training radiologists and other interpreters of radiologic studies 
to track and to determine their diagnostic accuracy and use of feature descriptors 
in the interpretation of radiologic examinations comprising: preparing a test set 
of images derived from radiologic examinations of patients; having the radiologist 
evaluate defined areas within the images to characterize an abnormality using 
accepted feature descriptors; and, reviewing the results of the evaluations to 
determine the radiologist 1 s ability to recognize abnormalities in the images and to 
properly characterize any abnormalities found using the feature descriptors thereby 
to improve the radiologist's accuracy of interpretation, reduce the variability in 
the use of standardized terminology, and train radiologists in understanding the 
relationship between feature descriptors and pathological entities found in 
radiologic examinations. 

27. A method for training radiologists and other interpreters of radiologic studies 
that tracks their use of feature descriptors and determines their diagnostic 
accuracy during interpretation of radiologic examinations and relates these 
performance data back to the findings in individual radiologic exams, comprising: 
preparing a test set of images derived from radiologic examinations of patients; 
having the radiologist evaluate defined abnormalites within the images to 
characterize the abnormalities using accepted feature descriptors and to assess the 
causes of the abnormalities; and, reviewing their evaluations of the radiologic 
examinations to determine their ability to accurately assess the causes of 
abnormalities and to consistently characterize abnormalities using the feature 
descriptors, thereby training radiologists to understand the relationship between 
feature descriptors and pathological entities found in radiologic examinations, 
improving the radiologist's accuracy of interpretation, and reducing variability in 
the use of standardized terminology. 
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